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FOREWORD 



Educators have often been criticized for viewing their role merely as dispensers of knowledge 
and showing a lack of concern with the application and relevance of this knowledge to their stu- 
dents' future lives. However, the general public has begun to accept the idea that the educational 
system has a responsibility to assist all individuals in making orderly transitions to the world of 
work. Additionally, during the last decade, a number of innovative activities, projects, and/or 
programs have been developed at the federal, state, and local level in which personnel are attempt- 
ing to link education and work. Among these programs and/or Droiects are the following examples: 
career education, Experience-Based Career Education (EBCE), Part D exemplary projects and other 
projects in vocational education, Title IV-C of the Elementary and Secondary Education Act 
(ESEA), Fund for the Improvement of Post Secondary Education (FIPSE), Title I and III of the 
Comprehensive Employment and Training Act (CETA), etc. 

Personnel associated with education and work programs are becoming increasingly aware of 
the need for information related to accountability and needed improvements. Many reports-some 
informal, some in the literature-indicate a wide and exciting variety of approaches to evaluating 
programs that link education and work. However, the need exists for practitioners to become 
acquainted with evaluation ideas and materials available for particular situations. 

Recognizing these trends, the Education and Work Group of the National Institute of Educa- 
tion (NIE) contracted with the National Center for Research in Vocational Education (NCRVE) to 
develop the Career Education Measurement Series. The series includes five comprehensive "user 
oriented" handbooks intended to help local education personnel in measurement and evaluation. 
The handbooks in the series are: 

• Assessing Experiential Learning in Career Education 

• Career Education Measures: A Compendium of Evaluation Instruments 

• Improving the Accountability of Career Education Programs: Evaluation Guidelines and 
Checklists 

• A Guide for Improving Locally Developed Career Education Measures 

• Using Systematic Observation Techniques in Evaluating Career Education 

This handbook. Using Systematic Observation Techniques in Evaluating Career Education, 
is designed to provide practical guidelines for using observational techniques to take "readings" 
of the feelings, perceptions and reactions of those persons involved in or affected by your projects. 
Furthermore, the handbook provides specific observation techniques that are well accepted as pro- 
viding objective information for improving career education projects. 

The Center is particularly indebted to Dr. Ralph Kester, former specialist at The Center and 
now Chairman of the Education Department at Greenville College, Greenville, Illinois, who prepared 
this handbook. Special recognition should be given to Dr. N. L. McCaslin, project director, who 
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coordinated the project. Additionally, appreciation is extended to Ms. Cheryl Lowry who edited 
the handbook. A special note of thanks is extended to Dr. Ron Bucknam who originally conceived 
this handbook series and continued his involvement through their development as Project Officer 
for the National Institute of Education. 

Valuable advice in the original conceptualization of the handbook was received from an 
advisory committee composed of Dr. Robert Ebel, Michigan State University; Dr. Margaret Ferqueroi 
State Director of Career Tducation in Florida; and Ms. Deede Sharpe, Georgia Department of 
Education. 

In an attempt to make this handbook truly "user oriented," the National Center is indebted to 
Drs. Fred and Murial Frank who reviewed the first draft and made suggestions prior to the user trials. 
Additionally, credit is given to those career education practitioners who participated in the user 
trial prior to publication. With their valuable assistance, the utility of this handbook has been 
enhanced. These individuals included: 

Samuel W. Corsi, Jr., Board of Cooperative Educational Services, Castleton-on-Hudson, N.Y. 

Kenneth Drenth, Career Education Planning District No. 6, Sault St. Marie, Mich. 

Ray Ehlers, Lexington Senior High School, Lexington, Ky. 

Polly Friend, Marquette-Alger Intermediate School District, Marquette, Minn. 

D. S.Tackley, Beard of Cooperative Educational Services, Binghamton, N.Y. 

Finally, a special note of appreciation is extended to Ms. Marlene Linton who typed the 
manuscript of this publication. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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CHAPTER 1 



INTRODUCTION 



What does this 
handbook have 
to do with you? 



Our view of 
evaluation 



If you are like most career education project directors, you probably 
have neither the time nor the expertise necessary to conduct extensive 
evaluation studies of the activities you direct. Furthermore, you may 
believe that many evaluations do not tell the complete story about educa- 
tional projects. But despite that belief and your lack of time and exper- 
tise, you still must somehow find out what effects your project is having. 
That's why this handbook was written. 

The handbook has been designed to provide some practical guidelines 
as to how you can deal with the evaluation responsibilities you face as an 
educational leader. While not intended to make you an evaluation expert, 
it should enable you to use respected data collection methods that can 
help you take a "reading" of the feelings, perceptions, and reactions of 
those involved in or affected by your project. More specifically, it should 
help you: 

• Collect information that can help you make decisions about the 
most valuable and worthwhile aspects of the project so as to better 
respond to the needs of accountability, continuance decisions, 
and modifications. 

• Collect information that can help you identify problem areas 
before they become critical to the continuance or effectiveness 
of the project. 

• Discover influences and effects of the project that were not 
anticipated before the project began. 

• Use the descriptive information you collect so that, like numerical 
data, it contributes to a meaningful comprehensive picture of what 
happened, with what effect, and perhaps with some clues as to 
why it happened. 



Since evaluation has been defined in many ways, no attempt is made 
here to provide the "ultimate" definition. However, it is important that 
some focus be provided so that you can better understand how to use the 
guidelines provided in this handbook. 

Evaluation, for our purposes, is defined as the process of (1) identify- 
ing key aspects of a career education project, (2) collecting pertinent 
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information concerning those aspects, and (3) presenting the information 
so that it can help you and others decide whether the project is successful 
and whether any parts of it need Improvement. 



The need for 
alternative 
evaluation 
techniques 



Defining evaluation in this manner allows the use of a variety of tech- 
niques for collecting information upon which to make decisions about the 
project. When educators think of evaluation techniques, many times we 
envision questionnaires and mountains of statistics. Questionnaires can 
certainly be a part of evaluation, and statistics are essential to making 
sense out of numerical data. However, the ever-present lack of time and 
money will surely mean that many decisions you will make about your 
project will be based simply on what you have seen, heard, or even how 
you "feel" about certain parts of the project rather than on data gathered 
with questionnaires and then analyzed statistically. That's where this 
handbook can help you. By helping you sharpen your skills of observa- 
tion, it can make it possible for you to evaluate your project in ways that 
are both feasible and meaningful. 

We try to point out how you can increase the amount and quality of 
data you gather through whatever contact you may have with project 
personnel or activities. Aiso, some suggestions are provided as to how you 
can multiply your information by getting others associated with the proj- 
ect involved in collecting and sharing information with you. 



The data We will focus attention on three basic ways to collect information: 

collection (1) observing verbal and nonverbal behavior, (2) interviewing, and (3) 

techniques gathering data unobtrusively. When actually collecting data, you may use 

to be a mixture of these techniques without distinguishing between them at the 

discussed time. 



These three forms of data collection are complementary to other 
forms such as questionnaires, standardized tests, or paper and pencil 
surveys. But separating these three forms of data collection (observing, 
interviewing, and gathering data unobtrusively) helps pinpoint important 
things to consider about each of them. It should help you to decide when 
and how to use a particular combination of these techniques in obtaining 
certain kinds of information. 

How the There are six other chapters that follow this introductory chapter: 

handbook 

is organized • Chapter 2 provides some insight as to what can be evaluated. 

• Chapter 3 presents a number of ways for systematically collecting 
data by observing, interviewing, and gathering data unobtrusively. 

• Chapter 4 shares some ideas and insights to be considered when 
using the techniques discussed in Chapter 3. 
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• Chapter 5 diicusses ways of organizing, analyzing, and justifying 
the use of the data collected. 



• Chapter 6 gives some rather concrete suggestions and exercises to 
increase your ability to effectively observe and record events. 

• Chapter 7 presents a brief summary of the information contained 
in this handbook and nine checklists to help you both review the 
information and plan your systematic observations. 

In addition, the Selected Bibliography at the end of this handbook 
identifies over fifteen other resources that deel with systematic observa- 
tion techniques. 
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CHAPTER 2 
WHAT CAN BE EVALUATED? 



Introduction 



There have been quite a number of articles and book* directed 
specifically at the question: What can be evaluated in career education 
programs and projects? It is not our purpose to completely answer the 
question; we can only provide some insights that should prove useful as 
you try to find out what is happening in your career education project. 
We think these insights will be particularly useful in regard to the three 
major types of data collection techniques discussed in this book— observ- 
ing, interviewing, and gathering data unobtrusively. 



Anticipated and 

unanticipated 

outcomes 



We have previously defined evaluation as a process of (1) identifying 
key aspects of a career education project, (2) collecting pertinent infor- 
mation concerning those aspects, and (3) presenting the information so 
that it can help you and others decide whether the project is successful 
and whether parts of it need improvement. 

The information you will be collecting and presenting will largely 
concern the outcomes of your project or any portion of it. (Outcomes 
aren't considered to occur only at the end of a project; they can be 
defined incrementally so that they can occur at all logical points during 
the conduct of the project.) You will be trying to find out whether your 
project, or parts of it, accomplished the objectives you intended. How* 
ever, education projects almost always have outcomes that were not 
intended or anticipated. Any evaluation effort, therefore, must take into 
account both anticipated and unanticipated outcomes. If no attempt 
were made to look for unanticipated outcomes, they would simply go 
unnoticed or be so vaguely referred to in impressions from other data 
sources as to be of no value to you. Often an evaluation that focuses only 
on anticipated outcomes must come to the disappointing conclusion that 
nothing "significant" occurred even though there are strong feelings that 
something, either good or bad, did result from the project. 



Anticipated and 

unanticipated 

influences 



In addition to the outcomes, there are anticipated and unanticipated 
influences— influences that can affect the conduct and perhaps outcomes 
of the project. Evaluation procedures that result in the discovery of un- 
anticipated influences can be useful in providing explanations of why 
anticipated outcomes did not occur. 



Used conscientiously the systematic observation techniques dis- 
cussed in this handbook can greatly assist you in determining the effect of 
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anticipated influences and to what extant anticipated outcomes occured. 
They should also provide information leading to the discovery of un- 
anticipated outcomes and influences. 

To believe it unnec ess ary to look for the unanticipated or search for 
information as to why anticipated outcomes did not occur is to assume 
that the implementation cf a career education project is a mechanical 
process, with all the lews of human action and interaction quite well 
defined. Even people who work on innovations for cers don't have it 
that good. Therefore, until we have a much clearer understanding of 
human nature and the leerning process, it behooves us to use every means 
available to gather end analyze es much retrievable information as possible 
in directing and evaluating career education projects in schools. To not do 
so will retard the progress of understanding, as well es set ourselves up for 
some disappointing and frustrating times when we just have to sit and 
wonder about what happened. 



Whet oen be The anticipated aspects of any evaluation effort are the intended 

anticipated in outcomes and anticipated influences. It needs to be emphasized that 

an evaluation? when we speek of outcomes, we ere not referring only to those which 

occur at the end of the project. Outcomes or results may be identified 
at any point and with any group associated with the project 

In the area of intended outcomes, the project's activities and objec- 
tives should provide some direction. In the absence of clearly defined 
objectives, the following potential outcomes for career education as they 
relate to observing, interviewing, and gathering data unobtrusively might 
provide some direction for the evaluation to take initially. 

• The following outcomes are just a sample of potential behaviors that 
could be intended or unintended focal points and that could be 
particularly appropriate for m/ustion by otmrving v$rM *nd non* 
vrbtl b$h§vior: 

• Students' work habits in or outside the classroom 

• Students' productivity in or outside the classroom 

• Students' alienation or compatibility with work situations in or 
outside the classroom 

• Students' social behavior in work settings in or outside the 
classroom 

• The relations between students and teachers-classroom atmos- 
phere and work orientation 

• The manner in which work or career decisions or patterns are 
presented 
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• Teecheri' encouragement of racial, sexual, or socioeconomic 
stereotypes as they relate to vocational planning and preparation 

• Interactions among teachers concerning the role and process of 
teaching career education 

• The methods, duration, and intensity of teaching career education 

Other outcomes such as information about affective and/or cognitive 
outcomes can be best gathered through the use of techniques like 
interviews and surveys and include: 

• Students' self-awareness, exhibited by: (1) their recognition of 
their interests, aptitudes, talents, and achievements in relation to 
various work or career goals; (2) their undemanding of their own 
place in society; (3) their knowledge of the forces that influence 
their career development; and (4) their application of their values 
in decision-making situations 

• Students' awareness of and ability to understand or apply general 
educational skills to work situations outside the formal classroom 

• Students' awareness of and ability to understand or apply eco- 
nomic theory and practice to decisions about desired work roles 
and general aspects of the world of work 

• Students' decision-making skills concerning their work and career 
goals 

• Students' demonstration of skills needed to obtain employment 

• Students' understanding of the concepts of job, work, occupation, 
vocation, and career 

• Students' application of various information (e.g., self-knowledge, 
specifications of the occupation or work role, culture or life-style 
of the occupational role) when making a work or career decision 

Still other outcomes sought through career education activities can 
be assessed through less direct methods such as unobtrusive mmsurm 
and include: 

• Teachers' ability to incorporate career education concepts in their 
classroom (e.g., bulletin boards, lesson plans, displays) 

• Teachers' professional development act.vities related to career 
education responsibilities 

• Teachers' efforts to update<%areer education information pre- 
sented (e.g., new equipment used in the work setting, skill require- 
ments for entry level jobs) 



• Students' performance on standardized achievement tests 

• Students' respect for property (e.g., buildings, equipment) 

• Students' attendance at extracurricular career education activities 



In summary, 
what effects on 
the project 
itself -the content 
or procedures-can 
be identified? 



The list of outcomes for career education above may seem long and 
rather involved, but you would probably like to have the answers to these 
and many more outcomes. It is important that you anticipate the kinds 
of outcomes you are likely to want information about. Otherwise, you 
will have to wait for the questions to "appear" (or be imposed) and then 
search for some reasonable way of answering them. This handbook sug- 
gests that you devise as comprehensive a systematic data collection 
scheme as possible. Then, as the program operates, you can collect data 
by the methods discussed here in order to take "pictures" of the events, 
outcomes, and general activity associated with the program. From 
periodic analyses of such data you can gleen information that will give 
you tremendous help not only in answering the predetermined questions, 
but in discovering the unanticipated and in raising new questions. How 
this can be done and discussion concerning the tools and procedures you 
use to collect the data is precisely what is covered in subsequent chapters. 
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CHAPTER 3 
SYSTEMATIC OBSERVATION TECHNIQUES 



Introduction Our discussion of techniques will cover three basic methods: (1) ob- 

serving verbal and non-verbal behavior, (2) interviewing people, and 
(3) gathering data unobtrusively. The information provided for each of 
the three methods will be: 

• A definition of the method 

• Descriptions and illustrations of its use— that is, specific techniques 

• Possible applications ;o evaluating career education projects 

• A discussion of the confidence one can establish in its use 



Definition 

Observing is a primary and fundamental technique that undergirds 
all of the others discussed in this chapter. In fact, one could easily make 
the case that all forms of data collection are simply derivatives of obser- 
vation. With this in mind, what will be discussed in this section will be 
more in the area of personally watching or listening to an individual or 
group and accurately recording what was done or said. 

In the research and evaluation literature, some labels for observation 
techniques are: participant observation, enumeration, and interaction 
analysis. The role you play as an observer varies according to: (1) the 
extent to which you actually participate in group activities and (2) how 
aware members of the group are that you are observing them for evaluative 
purposes. 1 Figure 1 shows how you can expect to be perceived by the group, 
given various combinations of your degree of participation and the degree 
to which the members realize you are an observer. 

Specific Techniques 

As a project director, you are obviously part of the project. However, 
various groups with whom you work may view your actual participation 



1 Raymond L Gold, "Roles in Sociological Field Observations/' Social Forces, Vol. 32 (1958), 
pp. 217-223; and Frank W. Lutz and Lawrence lannaccone, Understanding Educational Organize- 
tions: A Field Study Approach (Columbus, Ohio: Charles E. Merrill Publishing Company, 1969). 



Observing 
people 



£8 



Figure 1 

PERCEPTION OF OBSERVER BY THOSE BEING OBSERVED 



High 



Participation in 
Group Activities 
by Observer 



Low 



Complete 


j Both Participant 


Participant 


I and Observer 






Limited 


l Complete 


Participant 


I Observer 



Awareness of Observer 
by Group Members 



High 



with them differently. Some individuals will immediately accept you. 
Others will not accept you as readily. As will be pointed out in Chapter 
4, how you are perceived by those you observe or how you gain access 
to observe them can be critical to how they behave. 

Your being the project director does not guarantee you immediate 
access to all parts of the project. You must use all your abilities to see, 
listen, think, feel, and record the events, actions, interactions, and other 
situational or circumstantial information. Such information can provide 
quite a complete picture as to just exactly what happened. The simple 
but ultimate test of the adequacy of an observational account is to ask 
whether a person reading it could then behaveappropriately as a member 
of the group observed, or whether he/she can anticipate and interpret 
what occurs in the group as well as its own members can (Wolcott, 1975). 

As far as the actual process of recording observations goes, you are 
limited only by what is to be observed and your ingenuity in devising a 
legitimate way of conscientiously and systematically recording the obser- 
vations. Exhibits 1 to 6 provide illustrations of forms or techniques you 
could devise to actually record data. 
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Exhibit 1 
OPEN-ENDED OBSERVATION 



Possible Use: This type of device could be used when little or no structure is being used to guide 
the observation, which is particularly appropriate when trying to determine the extent to which 
stuaents are developing work habits and attitudes. It should allow the observer to keep track of 
what students and teachers are saying and doing related to work habits and attitudes. As a result 
of the actual observations or dialogue, you could then react to the extent that work habits and 
attitudes are being established. 



Actual Observations 
or Dialogue 



Feelings or Impressions about 
What Was Done or Said 



(This section should contain only the 
account of what is actually observed 
or said. No judgments or evaluations 
should be listed in this side of the 
observational record.) 

e Students were all actively 
involved in designing posters 
related to possible careers. 



"That would be a terrible 
career, I can't stand to get 
my hands dirty." 

"I wish we would hurry up 
and get these posters done.' 



One student did not have 
any pictures and was asking 
others for some he could use. 



(This side should contain the emotional 
response or interpretation that the 
observer had about what he/she saw or 
heard.) 



The teacher had command of the students' 
attention and all were exhibiting very positive 
relationships with one another and each person 
seemed to be cooperating 

Students were free to express their opinions 
and were making judgments about how they 
would feel in particular careen. 

This student was not able to focus on the real 
purpose of the activity. She seemed to be only 
interested in complying with the teacher's direc- 
tions. 

This student was not prepared-I'm not sure how 
dependable his work habits might be. 



Exhibit 2 



INTERACTION ANALYSIS 2 



Possible Use: Interaction analysis is a way of describing what goes on in an environment or setting. 

Flanders' interaction analysis is one of the most widely known and researched techniques. The first 
step in using interaction analysis is to develop observation categories that represent typical be* 
haviors and to assign index {or code) numbers to each category. The second step is to observe 
behaviors and record them by index number. The third step is the arrangement of those numbers 
representing behavior categories into a matrix that will be useful as you try to interpret what went 
on. The fourth, and final step is interpretation. 

Let us assume that one of the purposes of a career education project is to encourage more interac- 
tion between students and employers and to discourage employer dominated situations. The use 
of interactive analysis could be used to monitor the extent to which the desired interaction is 
occurring. 

Step 1: OBSERVATION CATEGORIES - In developing observation categories, it will be neces- 
sary to define the categories to be mutually exclusive but, as a whole, to accommodate as many 
kinds of behavior as you are likely to find in your observations. The categories should be grouped 
in terms of the people exhibiting the behavior in the categories (i.e., student behaviors covered in 
categories 1-5, employer behavior in categories 7-10). It is important that you memorize the 
categories and their corresponding index numbers; otherwise it will be impossible for you to record 
your observations rapidly enough. An example of such an observation category system and the 
index numbers assigned to each could include the following: 

Index Number 



2 Also see: N. A. F landers, Analyzing Teaching Behavior (Reading, Massachusetts: Addison- 
Wesley, 1970); and Murray Melvin, "Field Methods and Techniques: The Action-Interaction Chart 
as a Research Tool," Human Organization, Vol. 12, No. 2 (Spring 1953), pp. 34-35. 



2 
3 
4 
5 
6 
7 




Student 
Behavior 



Students asking question 
Students responding to question 
Students initiating comment 
Students responding to comment 
Students making irrelevant noise 
Silence 

Employer dominating conversation 
Employer asking question 
Employer responding to question 
Employer commenting 




Employer 
Behavior 
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Exhibit 2 - Continued 



Step 2: OBSERVING AND RECORDING BEHAVIOR - Before beginning observations, you 
need to determine (a) how often you will use this technique, (b) how long you will spend observing 
during each session, and (c) how often will you sample specific behavior. In our example, this is 
the first of four observations of fifteen minutes each, recording specific behaviors every ten seconds. 
As you observe the situation, you would record the index number associated with the category of 
behavior that is occurring, making a record about every ten seconds. In this way you might end up 
with a listing of index numbers such as the following: 

6 

7 

7 

7 

1 

1 

1 

9 

9 

9 

3 

3 
10 
10 
10 

- and so forth until fifteen minutes have elapsed or until ninety 
observations have been made. 

Step 3: AR RANGEMENT OF OBSERVATIONS - The first activity of this step is to arrange 
your recorded index numbers in pairs. To do this the first two numbers are paired together. Then 
the second and third numbers are paired together. Next, the third and fourth numbers are paired 
together, and so forth. In our example, these pairs would be as follows: 

6- 7 

7- 7 
7-7 
7-1 
1-1 
1-1 
1-9 
9-9 
9-9 
9-3 
3-3 
3-10 

10-10 
10-10 

and so forth 
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Exhibit 2 - Continued 



The second activity of this step is to arrange the pairs of numbers in a matrix for interpretation. 
The matrix that results from our example could be displayed as follows: 



Student 
Behavior 



First Index 
Number of 
the Pair 



Employer 
Behavior 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



Second Index Number of the Pair 

Student Employer 
Behavior Behavior 

3 4 5 S 7 8 9 



10 



Silence 



Silence 



Begin with the first pair of numbers. Find the first number of that pair (in our example it is 6) and 
locate it in the column of index numbers to the left of the matrix. Follow the row opposite that 
number across the matrix to the column that is headed by the second number of the pair (in our 
example, 7). Make a tally mark in that cell of the matrix. (Tally marks have been placed for the 
first two pairs of our example in the matrix below). 



Second Index Number of the Mr 



Student 
Behavior 



First Index 
Number of 
the Pair 



Employer 
Behavior 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



Student 
Behavior 



Employer 
Behavior 



















































































































1 




















1 





































































Silanoa 



SilWK* 
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Exhibit 2 - Continued 



Continue the process until you have dealt with all pairs of your index numbers. Then count the 
tally marks in each cell, replacing the marks with a number that represents the total of tally marks 
in each. (The matrix below has been completed for all index numbers in our example.) 



Student 
Behavior 



First Index 
Number of 
the Pair 



Employer 
Behavior 



1 
2 
3 
4 
5 
6 
7 

a 

9 

10 



Second Index Number of the Pair 
Student Employer 
Behavior Behavior 
23456789 



10 



2 
















1 




























1 














1 






















































1 








1 












2 
































1 












2 
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Silenca 



Siltnoa 



Step 4: INTERPRETATION - In our example, we wanted to encourage more interaction between 
students and employers. Furthermore, we wanted to discourage employer dominated situations. If 
this were to occur we would expect to have a lot of cells filled in the upper right and lower left 
quadrants of the matrix because totals in these cells reflect behavior that involved movement from 
student behavior to employer behavior and vice versa. Therefore, the totals in cells in the upper 
right quadrant reflect student-employer interactions and those in the lower left quadrant reflect 
employer-student interactions. Also, we would expect fewer cells to be filled in the lower right 
quadrant. Examination of the last matrix presented in Step 3 suggests that we did not achieve our 
goal for this first observation. However, you need to repeat this type of observation three more 
times, as previously planned to actually have any confidence in the results. In addition to provid- 
ing effectiveness data about the career education project, this technique also provides feedback to 
the teacher regarding performance in achieving the desired goals of the project. 
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ISOLATED BEHAVIOR ENUMERATION 



Possible Use: This form of observation could be used when you have rather specific singular be- 
haviors you desire to enumerate. The behaviors must be well defined. 

If you desire, for instance, to determine how often students or the teacher of a class refer to certain 
concepts related to the world of work, you could simply define the categories to be observed, sys- 
tematically record the frequency and duration of each occurrence of each category of behavior, and 
then compute the ratio of frequency to duration. Among the categories you may have defined to 
use in our examples concerning references to certain concepts related to the world of work may be: 

(1) mentioning the terms work or job in a context other than classroom assignment work, and 

(2) mention of specific careers or job titles. 

Tims Span Frequency 
Behavioral Category Frequency Ofawrvsd Minutes 



1 . Mentions the term 

work or job other Teacher 
than that related Student 
to class assign- 
ment work 



2. 


Mentions specific 
careers or job 
titles 


Teacher 
Student 


3. 




Teacher 
Student 


4. 




Teacher 
Student 


5. 




Teacher 
Student 


6. 




Teacher 



Student 



25 

16 



Exhibit 4 
MAPPING ACTIONS 



Possible Use: Mapping actions is an observation approach to recording the movement of all 
or certain persons in a group. 

This observation technique could be used, for example, to determine the extent to which 
students are using a career center in a classroom. You could "code" the person or persons on 
which you wish to keep track in a manner similar to the following: 

x Bob © Bill 

0 Sally E June 

(x) Jim 

• Teacher 



You could also code certain types of actions, but it gets rather difficult to keep track of all of 
this after a while. The rule is "keep it simple." 

TMchtr'i Dttk 
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Exhibit 6 



CRITICAL EVENT RECORDING 



Potiibto Use: Critical events have been defined in numerous ways, Ont dtfinition is simply any 
tvtnt or incident that may hava had soma potential influtnca on tha program. Ona convaniant 
aspact of critical event recording is that its use is not limited to the project director. Almost any 
actor within a program could be involved in recording such events. 

Each recording of a critical event should contain the basic information of who did (or said) what 
when, where, to whom (or in the presence of whom), with what apparent reection. Later, the event 
cm be followed-up as to any "ripple" effects or actions taken. It can be used effectively to monitor 
student problems and attitudes. 

Sample 1: 



Critical Event Form 

Recorder J.Jonm Dat a 4M/77 Time 2:30 pm 

Plac e 9th grede elgebre dess Context 

Who was involved? Bill Smith, a fresnmen; end Mr. Brown, a freshmen sfgsbre teecher 
What was done or said to whom (or in whose presence)? 

Bill told Mr. Brown thet he couldn't see why someone should teke elgebre end the 

rmt of the students listened, 

What were the reactions? Mr. Brown seid thet it wes reouired by the school boerd. 
What action was taken? Mr. Brown did not setts fy Bill's question end Bill subeeouentiy 
did not listen to the remeinder of the cless . 



What were your impressions concerning its potential effect on the program (positive or 
negative)? 

Mr. Brown lost en excellent opportunity to show how elgebre is releted to en eree 
of special interest to Bill. 
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Exhibit 6 - Continued 



Sample 2: 

Critical Event Report 3 



Description of Actors, Activity, 
Decision, or Event 



Anticipated (Why?) 



Unanticipated (Why?) 



Likely Consequences: 



Use the reverse side for additional comments if necessary. 



3 Also see: Kathryn A. Hecht, 'The Development of a Chronology of Critical Events," a paper 
prepared for the Annual Meeting of the American Educational Research Association, San Francisco, 
California, 1976. 
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Exhibit 6 



DIARY KEEPING 4 

Possible Use: This form of collecting observation data amounts to having students, teachers, or 
others associated with your program simply keep a log or diary of the events of each day. Such 
logs can be very useful in determining the amount and "pulse" of activity and can result in the 
identification of some interesting patterns of behavior. Also, if systematically and continuously 
recorded and graphed over time, it could be used to predict lulls in program activity. 

Students, teechers, or others could keep logs or diaries similar to that below. 

Weakly Report 
Neme 

Date Project 

Total Hours 

Teaching — 

Conferences (specify) ^ ^ 

Preparation 

Administrative duties _ 

Other duties 

Comments on specific things done or actions taken in reference to the career education project. 
Things done: 

Actions taken: 



4 Also see: George H. Axinn and Nancy W. Axinn, "The Indigenous Observer Diary-Keeper: A 
Methodological Note." Human Organization, Vol. 28, No. 1 (Spring 1969), pp. 78-86. 
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Pcmibl* Applications 



Observation techniques can be used in some manner with both the 
anticipated and unanticipated aspects of evaluation. The following is a 
list of possible areas where observation techniques could be used. 

• Determining unsolicited perceptions, feelings, and attitudes toward 
a career education project 

• Recording specific responses of acceptance or resistance to a project 

• Recording student behavioral outcomes or other responses to activ- 
ities related to a project 

• Keeping track of actions, reactions, and other occurrences from the 
perspective of the school personnel, and employers of sponsors 

This list, although in no way complete, should provide some stimulus 
for thinking of other areas where the application of direct observation 
techniques would be useful. One point to remember is that you, as a 
project director, are naturally going to be observing to a certain extent 
anyway. Therefore, you might as well systematize the process so as to 
make the information more useful to the total evaluation plan. Another 
aspect of the observation process is to assure confidence in the obeerve- 
tions you make and record which is discussed next. 

Assuring Confidence 

One of the major problems that people have with the collection 
and use of direct observation data seems to be that they feel such data are 
potentially biased and, therefore, neither valid nor reliable. That is, they 
don't believe that the methods used to collect the data are controlled 
enough to ( 1) actually take a measure of what it is they are supposed to 
measure or (2) be dependable. This is not necessarily so. Observation 
techniques are no more or less inherently subject to bias than any other 
data collection methods-including those used in experimental design. 
Observation data can be collected and verified in such a manner as to 
render them both valid and reliable. Below are some points to remember 
and put into action in order to insure greater confidence in the informa- 
tion you gather. 

• Do not become too self-conscious about sometimes having to 
pretend to participate on occasion in order to gain access to per- 
tinent data. Being too self-conscious could alert the members of 
the observed group and cause them to react differently than they 
otherwise would. 

e Maintain objectivity about the project. As a project director, you 
may spend so much time participating that you become more 
enamored by or involved in the actual participation in the activ- 
ities than by observing what is going on. 
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• Remember that the observations you make are dependent on how 
the group perceives you. Therefore, the observations you make will 
be limited in this respect. 



• Keep in mind that a person's position (even yours as an observer) in a 
group determines to some extent how he/she perceives any group 
situation. 

• Be careful to act in a way that is not threatening to the members 
of the group you are observing. Any perceived threat could invali- 
date or lower the reliability of the data you gather under such 
circumstances. 

• Always attempt to gain corroboration of information and check the 
reliability of observations made from the testimonies of others who 
were involved in the incidents on which you have information. A 
good rule is do not use information that cannot be corroborated 
by at least one other testimony or set of data. 

• Avoid misinterpretation. The picture you obtain using observation 
techniques is similar to a series of snapshots of a fast-moving event. 
From these "snapshots," attempts are then made to reconstruct the 
event. If the "snapshots" have not captured the important shifts 
of activity or dialogue, there is a danger of reading into the overall 
picture actions and words that may not have occurred. However, 
by checking reactions to various events or activities at different 
times (time and event sampling) and by obtaining corroborating 
testimony about the observed and interpreted data, you can be quite 
confident that an adequate description of what did occur has been 
constructed. 

• Do not use diaries or critical event records from untrained persons 
as primary data sources. This is because of the higher probability 
of biased information resulting from a combination of inexperience 
and vested interest in the activity being observed by untrained 
individuals. Use such information for corroboration, supporting, or 
elaboration purposes. 

• The key to properly analyzing isolated behavior or interaction is 
making precise category definitions and very carefully recording 
observations according to those categories. If you have assigned 
more than one person to observe the same kinds of events or ac- 
tivities, it is important that you review their observations in order 
to make sure that they are using the same categories and working 
the same behaviors under those categories. These reviews are neces- 
sary to increase confidence in the data gathered. 

These procedures are no more cumbersome or time consuming than 
those steps or procedures that should be employed in order to confirm 
and insure the validity or reliability of data gathered in other ways. Also, 
it can be argued that data gathered through direct observation, if 
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corroborated, is probably the most valid and reliable data that you can 
obtain because they are generally primary source data. In other words, 
the data come from the actual event rather than from someone's after-the- 
fact interpretation of the event. 



The technique of talking with people, or interviewing, is one of the 
most common forms of collecting information. Probably because of its 
common use, many people believe that interviewing is merely asking 
questions of people. Asking questions is naturally a part of the interview- 
ing process, but the success of an interview does not lie in simply asking 
questions. Obtaining information through interviews requires careful 
planning and control. 

Specific Techniques 

The interview situation is set in a milieu of personal, interpersonal, 
and situational factors. Numerous books have been written on how one 
goes about doing interviews. In this brief discussion we can only intro- 
duce you to some of the major issues. Interviewing, like many of the 
other systematic observation techniques, is an art-science. That is, there 
are some basic scientific principles that can guide the interviewer and give 
him/her reasonable assurance that the information gathered is reliable; 
however, the specific approaches used are an artful combination of some 
basic ways to glean information from other people. Once the basics are 
known, coached practice is the main road to success. 

Interviews may vary from informal moments when you have the 
opportunity to chat with some of the people involved in thl project to 
a carefully nurtured relationship with a reliable person particularly knowl- 
edgeable about the situation in question. Because of your many oppor- 
tunities to formally and informally conduct interviews, you should 
a/ways have in mind some unanswered questions that fit into particular 
parts of the picture you are constructing to describe the situation under 
investigation. 

Persons associated with the group being studied and with whom 
you have been able to establish good rapport and a confidential relation- 
ship can be of great value to you as you gather information. Generally, 
the best persons to use for such purposes are those whose positions in 
the group enable them to view a broad spectrum of activity and comment. 
You would also want the persons you use to be reasonably articulate 
about the situations and events in which they become involved. 

At the beginning of a project, they can be used to provide you a 
quick overview of the situation. During the project, they can be used 
to identify areas that need further exploration or to provide or confirm 
< information from other sources. Informed persons can also be useful at 
the end of a program when you are putting together certain missing 
"links" and trying to understand what has happened. 



Interviewing 



Definition 
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Using two or three informed persons will help you establish the 
reliability of the data gathered. If at all possible, it is better not to let 
these persons know of each other-or at least not of the role they are play- 
ing for you in providing information about events associated with the 
project. I n this way, they will not feel compelled to have similar stories. 

It must be remembered that data gathered from informed persons are 
still a mixture of primary source, secondary source, and hearsay informa- 
tion. It will be imperative that you carefully analyze and evaluate the 
information given so as not to attach greater significance to what was 
said than is warranted. 

The discussion that follows divided interviewing techniques into two 
basic types: unstructured and structured. 

Unstructured Interviews. The unstructured interview is probably 
the most artful of any of the interviewing techniques. It is a purposeful 
discussion between you and another person, but without specific pre- 
structured questions. The emphasis is on listening as you discuss things 
that are important to the interviewee within the context of the general 
concern or focus of the career education project. 

The intent of the unstructured interview is to elicit from the person 
being interviewed a highly personal and detailed account of particular 
events or sentiments related to the career education project. To do this 
you must consider a number of complex influences that bear upon the 
interview situation. 

The manner in which you approach or are introduced to the inter- 
viewee is important. Remember, interviews are an intrusion on the time 
and work of the interviewee. You must make sure the interviewee is con- 
vinced that you believe his/her information is important to the project. 
It is also important to establish a good relationship— one of trust and 
confidence— between yourself and the interviewee. All of this is difficult 
to establish if the interviewee is rudely interrupted or perceives that he/ 
she is being coerced into being interviewed. 

Once the initial contact and relationship has been established, your 
work has just begun. You must continue reinforcing the relationship 
while gently and diplomatically probing the interviewee about specifics 
of the events or sentiments that are brought up. 

There are a number of ways to lead, guide, and probe without 
actually directing what the interviewee says. The following are generally 
useful steps: 

e Find out what the interviewee considers important by asking him/her 
a question about the general event or situation you are investigating. 

e Move from sentiments about this situation or event to associated 
interpersonal actions. 
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• After this general overview from the interviewee, get him/her to start 
at the beginning and go step by step through particular situations or 
events. 

• You can prompt or probe the interviewee in a variety of ways in 
order to clarify the facts as he/she remembers them. 

• One way is to "play dumb/' pretending not to understand certain 
inferences the interviewee makes. 

• Another tactic might be to take a skeptical air about various 
statements or actions the interviewee attributes to others. In this 
manner, you may be able to see if he/she is exaggerating or passing 
on hearsay. Of course, in using these tactics, you must not lose 
confidence or trust of the interviewee. Never get into an argument 
and, in general, do not interrupt the interviewee. 

• Every so often you should restate what you perceive the inter* 
viewee has said. In this way, you can confirm your understanding 
and/or redirect the conversation if the interviewee is getting off on 
a tangential issue. 

• By using some of the techniques discussed above, you should be able 
to pin down events, people, and times. This information will be 
valuable to you as you confirm the interviewee's account and further 
investigate the situation you are studying, perhaps by interviewing 
the persons referred to by the first interviewee. 

• If the interviewee is reluctant to discuss a particular event, one tactic 
would be to set up a similar but hypothetical situation and have 
him/her relate the comment to that. Sometimes such a tactic will 
allow the interviewee to discuss some feelings without divulging what 
he/she believes might be confidential. 

• A final point to remember is to give assurance to the person inter- 
viewed that you will hold in strict confidence any comment he/she 
shared with you. Be sure to keep that commitment. 

The discussion above, although extremely brief in comparison with what 
is available on the subject, should provide at least some basic guidelines 
for you to follow in conducting interviews. 

Structured Interviews. The structured interview is basically a ques- 
tionnaire whose questions are asked by the interviewer either in person or 
over the telephone. The structured interview technique differs in two 
ways from the questionnaire or survey technique. In the first place, the 
questions are personally administered by the interviewer. Secondly, 
although some of the questions may have categories for response, there 
will most often be some open-ended questions about which the inter- 
viewee is encouraged to share his/her knowledge or opinions. The struc- 
tured interview has the advantage of being able to pick up a great deal of 
information in a short period of time. 
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The questions you ask in structured interviews must be devised with 
an eye to the "big picture" of your project. This does not mean that you 
ask only global questions about the project. It does mean that you should 
keep in mind the project as a whole as you decide what it is you want to 
find out in the interviews and structure the questions accordingly. 

It is important that the questions be written in a manner that will 
elicit a valid response. To elicit valid responses, they must be written very 
precisely but also in the language and idioms of the respondents. They 
should be pilot tested with a few of the respondents and revised ac- 
cordingly before the questionnaire is finally used. If you have several 
open-ended questions it may be worthwhile to have the respondents 
view the questions before you conduct the structured interview). This 
will allow the respondents to contemplate the questions and perhaps be 
able to provide you a more thorough response. 

One appropriate time to use structured interviews is when you are 
interested in obtaining some fairly specific responses to a number of 
questions after other observation and unstructured interviews. Perhaps, 
with a broader audience, you will be able to confirm some points about 
events or sentiments or find out whether some new areas of interest or 
concern, uncovered through isolated observation or unstructured inter- 
views, are of enough general interest to pursue. 

Possible Applications 

In addition to using interviews for the purposes already mentioned 
(to clarify, confirm, or otherwise illustrate), you can also use interviews 
to gather a variety of other data. By talking to people on a regular and 
somewhat systematized basis, you can even gather needs a s se ssment or 
other contextual data. Interviewing can also be used to obtain opinions 
about certain career education materials or ideas that are being considered 
for use. One of the best uses for the interview is to obtain some feedback 
from students, teachers, parents, employers, or others about their reac- 
tions to particular career education activities of which they may have 
been a part. 

The interview can also be used to gather information about the 
impact of certain project activities-most likely their impact on opinions 
or attitudes rather than on behavior or knowledge. However, some 
confirmation or checking of their impact on behavior could be made 
through the use of interviews by asking persons associated with a par- 
ticular activity what they perceived the impact to be, or, the extent to 
which they observed certain anticipated outcomes to have occurred. 

Assuring Confidence 

The first rule of thumb is that interview data tend to be strongest 
when combined with data from other sources collected through other 
means. If they reasonably correspond with information gathered through 
other means and sources, the data from interviews can act both as a 
reliability check on other data and as supporting or clarifying data. 
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The validity of inttrvitw data can be judged on whether the infor- 
mation it relevant to the intent of the question and accurate from the 
respondent's perspective. If an interviewee misinterprets a question or 
feels he/she cannot openly share his/her knowledge or opinions it will 
mean that the information he/she gives you is invalid. You, as the inter- 
viewer, also have some responsibility here. You must be alert to what 
the interviewee is saying and record the proper response according to your 
categories of response or record as accurately as possible what it was the 
interviewee said or did in response to an open-ended question. Data 
gathered through the use of the interview can be checked for accuracy by 
others associated with the career education project. If there are discre- 
pancies or inconsistencies between the data presented and the perceptions 
of those studied, several questions must be considered. For example: Are 
the discrepancies and inconsistencies the fault of misinterpreted questions 
or misinterpreted answers? Or are they simply a reflection of the actual 
ambiguity of the situation investigated? These are difficult questions but 
ones that may need to be dealt with when you attempt to validate the 
interview data. 

The reliability of interview data has two avenues of concern. If 
there are several interviewers attempting to obtain comparable data with 
the same set of structured questions, you'll want to know whether they 
are recording the answers in the same way. Perhaps the best way to check 
this is to sit in on some interviews with each of them and discuss them 
afterward. The other question of reliability is really tied to that of 
validity and has to do with the trustworthiness of the interviewee. That 
is, is there corroborating evidence to support the case that the interviewee 
generally provided truthful information? 



Gathering Definition 
data 

unobtrusively Unobtrusive methods are those data collection methods that do not 

require direct contact with individuals. Archeologists are probably the 
most adept in using unobtrusive methods to piece together the culture of 
a particular situation. Police detectives are another set of investigative 
persons who rely quite heavily on evidence without the individuals in- 
volved being present. Four basic types of unobtrusive methods will be 
discussed here: physical traces, artifacts, records, and indices. 



Specific Techniques 

Physical Traces. Physical traces are observable signs of the use of 
objects or space in a particular setting. Some examples of physical traces 
in relation to career education projects are physical evidence of wear on 
books, films and film projectors, areas of the room (e.g., worn carpet or 
flooring around a particular activity area), and computer printout scraps 
in the waste basket next to a terminal. 

Artifacts. Artifacts are objects or material that can be identified as 
resulting from human activity. Examples of artifacts in career education 
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settings are lesson plant, curriculum units, and materiels or things de- 
veloped in conjunction with student projects (e.g., objects made, written 
materiel, or some other evidence of student work). 

Records. Records such as minutes of meetings, correspondence 
files, statistics or demographic data, information compiled for such files 
as accumulative student records, a lesson plan book, or even e grade book 
can contain important information that can become e pert of the total 
picture you are trying to arrange. 

Indices. Indices are what some people refer to es "manufactured" 
data in that they are contrived comparisons of statistical data such as an 
index of the percentage of persons with previous work experience in each 
of several experience-besed projects. The ratio of the number of boys to 
girls in selected classrooms, IQ scores, socioeconomic ratings, end other 
such ratios are alio indices. 

Possible Applications 

The activities of carter education projects could result in some 
evidence of use of equipment or materials end gen e r a te some materials 
or objects that reflect various aspects of *he project. If there is a career 
information center, several evidences of use end artifacts may be evident. 
A computer terminal will have automatically recorded all uses. Even the 
waste basket near the computer terminel might provide evidence of how 
and for what purposes students were using it. Books, files, pamphlets, 
and other material, if used, will have signs of use, e.g., "d og eared** pages, 
files out of piece, papers in files shuffled, end pamphlets in locations 
other than where they were originally placed. You might sven be able to 
tell where students spend the most time in the career information center 
by observing where the most litter or dirt on the floor or carpet is located. 
These, plus numerous other logically connected evidences of use, could 
become insightful pieces of information, which, when put together with 
other more direct evidence, would provide a more complete description 
of what occurred. 

Career education projects may also have students produce something. 
It might be as simple as responding to a set of vocational interest ques- 
tions or as complex as manufacturing some item to demonstrate some set 
of manual or cognitive competencies. Teechers and administrators also 
generate artifacts of programs in the form of lesson plans, curriculum 
units, and records of planning and accomplishment. All of these can 
provide insight into the functioning of a project and thus should be con- 
sidered a legitimate, important, and integral part of an evaluation. 

Assuring Confidence in Unobtrusive Deta 

Relying on evidence of use many times requires a certain level of 
inference, conjecture, or assumption. One can have confidence in its 
ability to provide leads and corroborate other self-report or observation 
data, but it is not often that such evidence alone is sufficient. Materials 
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or objects produced by students or teechers, on the other hend, ere con- 
crete evidence of activity. A difficulty with relying on meteriels end 
objects produced is thet there mey be little evidence et to who per- 
ticipeted end over whet period of time. However, if artifacts ere used es 
partial data sources of en evaluation and can be logically or empirically 
linked to other date, they can add tremendously to the concreteness of 
any evaluation report. 

It is desirable to determine the reliebility of the unobtrusive data 
gathered by meesuring the outcome in other ways. For example, con* 
fidence in information supplied by records and indices comes primarily 
as a result of the confidence that one can assure in the accuracy and 
completeness of the records being used. Some corroboration of record 
data is important-checking the record with persons who were involved 
in the informetion recorded or with other records that parallel the same 
events. 

All in all, data collected unobtrusively can provide insightful leads, 
confirm other data collected, creete new data that link existing sets, and 
otherwise add significant and relevant information to an evaluation. 



In summary, how 
can systematic 
o b s ervat ion 
techniques best 
be used? 



One way to talk about the types of information you might wish to 
collect is in terms of four categories: (1) how often certain events, be- 
haviors, or situations occur; (2) descriptions of incidents or histories; 
(3) social norms and rules of conduct; and (4) personally held or ex- 
pressed beliefs and values. 

Figure 2 provides a quick reference to the various techniques dis- 
cussed in this chapter and some judgment es to which of the four cate- 
gories of information the techniques are best suited. This figure should 
help you to recall the various techniques and give you a point from which 
to begin your selection and use of these techniques in your present 
responsibility. 
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Figure 2 



USE OF DATA COLLECTION METHODS FOR 
TYPES OF INFORMATION DESIRED 3 



Methods of Obtaining Observational Data 
Information 



Types Using Unobtrusive 

Observing I nterviewing Methods 



Frequency 


Enumeration 


distributions 


Isolated behavior 




Interaction 




analysis 


Incidents and 


Open-ended 


histories 


Diary keeping 




Critical event 




recording 


Social norms 


Combinations 


and rules 


of all 


Personal 


Open-ended 


beliefs and 


Diary keeping 


values 


Critical event 




recording 



Generally 
inadequate and 

inefficient 



Physical traces 

Artifacts 

Indices 



Structured 
Unstructured 
Informed persons 



Records 
Artifacts 



Combinations 
of all 



Records 



Combinations 
of all 



Records 
Artifacts 



5 This chart is based on a similar analysis by Morris Zelditch, Jr., as reported in George J. McCall 
and J. L. Simmons, Issues in Participant Observation (Reading, Massachusetts: Addison-Wesley 
Publishing Co., 1969). 
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CHAPTER 4 



THINGS TO CONSIDER WHEN USING 
SYSTEMATIC OBSERVATION TECHNIQUES 



Introduction When you are observing a career education project, you should take 

into account the many things that may potentially affect the outcomes 
of your observation. For that reason, it is important that a number of 
issues be thought out, considered, and dealt with prior to and throughout 
your project. Chief among them are: 

• Various issues concerning the nature of data collection and your 
role 

• The manner with which you introduce and maintain contact with 
the various groups related to the program 

• Sources of tension that may exist as a result of conducting the 
program and evaluating it 

• The importance of training and practice prior to observing the 
program 

• Checking reactions to events and activities at various times and 
places (time and event sampling) 



Data collection At the beginning of any evaluation, the question should be asked: 

issues Is there any violation of the general right to privacy or a possible threat to 

the subjects' (people you are collecting information about) physical or 
mental health? How the data are to be used is critical in this issue. If the 
information is to be used to evaluate and report on the performance of 
individuals or an identifiable group, it is generally agreed among evaluators 
that those evaluated should be made awa-e of why the information is 
being collected and how it is to be used. If, on the other hand, the infor- 
mation will not be reported in a manner that identifies individuals or 
groups or makes their identification possible, permission is generally 
necessary only *rom a legitimate supervisor of the individual or group. 

The decision to choose a secret or open role is associated to some 
extent with the previous issue concerning the rights of subjects. In some 
cases, you may choose to conceal your identity. The use of unobtrusive 
measures for collecting data makes remaining anonymous quite easy. 
However, many people find a secret position untenable. Some reports 
indicate that they begin to feel guilty about collecting information about 
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people, especially if they know the people very well or art involved with 
the situition to any extent. This guilt may toad to inhibitions about 
collecting certain kinds of information and can destroy their et.sctlveneer 
In addition, people may share a co nsid erable amount of their personal 
lives with an observer perceived to be Interested and sympathetic This 
personal knowledgs may become burdensome for the observer end give 
him/her the uncomfortable feeling that he/she is "putting on" the indi- 
viduals simp!y to gsther information about the project. Or he/she may 
feel obligated to become involved with others' personal problems end 
thereby jeopardize his/her objective position. 

For these reesons, it is generally easier to acknowledge your purpose 
for observation openly to the individuals a sso ciated with the protect. 
This means sharing the basic intent, process, end duration of the obser- 
vation activity. 

Although there ere numerous other issues that you oould consider in 
observing your project, the final one mentioned here is the accuracy of 
your collection and reporting. Accuracy is something that you owe the 
persons being observed and the project as a whole. 

You must also be concerned with the issues of validity and relia- 
bility. One way of checking validity and reliability would be to have 
representatives from the groups being studied essess the accuracy of the 
written description and i nt erp r e tat ion of the information before it is 
r e po rte d and used for improving the project 

The point of this brief discussion concerning issues you may fate is 
that a project directo* must fulfill many roles, some of which conflict 
at times. Therefore, some tough decisions will need to be made to protect 
the rights of individuels yet at the seme time move the project a heed and 
provide an accurate record of what tra n spi r ed. 



Being A point that is often taken for granted is your introduction and 

accepted identification as an observer. Even though your introductory identifica- 

tion is as the project director, you may choose to fully assume an observer 
role and hide your project director identity. If vou choose this course, it 
is essential to carefully place yourself into this new role. Some steps you 
should follow when initiating an observer role are: 

e Talk with as many participants as possible 

• Initiate non-verbal actions that increase your credibility as a group 
member (e.g., during a presentation, be seated with the participants 
rather than alone in the back of the room) 

e Do not shift back into the project director role (e.g., clarifying re- 
sponses to questions posed to others while assuming the observer 
role). 
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Btfort you choose to play the obmrm role, several issues should be kept 
in mind. First, if you art wall known at tha projact director, it will ba 
difficult to Map into tha un identified obaarvar rota. In a lerger projact 
whara taachan and othan involvad In tha projact ara not awara of your 
projact diractor identity, tha transition will ba easier. Secondly, it It 
important not to confuse paopla at to tha rota you ara playing by chang- 
ing your rola midstreem. Third, it it normal behavior for indlvidualt 
in an attablithad group to ba curious and pottibly tvan tutpiciout of 
partont who coma into thair mktatt aithar unannounced or vaguaty an- 
nounced. Such an unoartain relationship could aatily invalidate data 
collaction or stifla or changa tha activities to ba obaarvad. 

In ordar to allaviatt anxiaty in systamatic observation situations, 
it is important that your entry ba through tha recognized social protocol. 
This might ba through tha principal of a school, tha superintendent 
of a district, tha chairparson of a particular dapartmant, or a particular 
taachar. Again, tvan if you ara somawhat known by tha paopla but do 
not saa tham continuously, it is important to follow soma form of tound 
protocol to that thay do not gat tha faaling that thay ara balng tnoopad 
on or harastad. It it important that tha parsons in tha group do not faal 
coarcad into accapting you. Tharafors, for instance, you must ba caraful 
that tha desires of the department haad(s) are not overridden by the 
principal or superintendent. 

Overall, you must be very careful at the outset and then continually 
check to determine whether you are being a cce p ted by the group for the 
purposes initially outlined to them. One thing you should prepare your- 
self for is that in times of conflict the members of a group may find it 
easy and advantageous to blame thair problem (s) on the project on tha 
activity, or even on you. Such an occurrence will call for soma extremely 
diplomatic action on your part in order to reestablish your position with 
the group. 



Other sources From reading soma of the other material in this handbook, you may 

of tension have decided that doing evaluation by using systematic observation tech- 

niques can be somewhat hazardous. We are not trying to scare you off 
but to establish an awareness and sensitivity to evaluation work among 
individuals and groups. If you meet people easily and are aware of the 
variety and intensity of interpersonal relationships, you should be rela- 
tively at ease in using the systematic observation approaches suggested 
in this handbook. 

The intent of using systematic observation techniques is to record 
what actually occurred in a particular event or situation. To do this 
requires utmost attention to all that is occurring. Such activity is intense 
and exhausting. With practice, you can pace yourself and determine how 
many observations should be made and how long and frequent they 
should be to maintain optimum effectiveness. You will also become more 
adept at interviewing so that you don't spend inordinate amounts of time 
on tangential details or obviously irrelevant discussion. 




4 



In addition to the strenuous physical and mental effort, other 
critical aspects of working with groups of people are the political and 
interpersonal relationships that can inadvertently throw you off the track 
or bias your judgment. Knowing this does not always alleviate the prob- 
lems that may arise. However, in general, if you know the political and 
interpersonal factors of a group, you are less likely to make serious mis* 
takes simply because you talked to the wrong person at the wrong time 
or said the wrong thing to some individual. Your success will be largely 
dependent upon whether people feel that you can be trusted with the 
information they volunteer or you observed, as well as with information 
that may have been gathered without their specific and direct knowledge. 

Training end The effective usa of systematic observation requires considerable 

practice practice. It is practice that seesons end forms the raw material of knowh 

edge into something usable for career education program improvement. 

Without some planned and controlled practice, you get yourself into 
some very embarrassing and personally harmful situations. For example, 
one article 6 reports that a student doing a study about coffee break 
activity in a county office created a minor political crisis between the 
office and a local taxpayers association. The association obtained some of 
the information and in ter preted it as indicating that county employees 
were "under-worked and over-coffeed." In another situation, a rookie 
investigator concluded that the actual activities of a particular service club 
in a small town was "bulling and boozing/' Needless to say, the members 
did not appreciate this sort of remark, and let the student, professor, and 
administration of the college know about it The results were harmful 
for all concerned. 

What we are saying, then, is that a person must have practice and 
perhaps even some supervision experience in real settings in order to 
develop, modify, and integrate a style of observation with which he/she 
is comfortable and which facilitates completion of the activity without 
harmful effects. Although it is relatively difficult for a project director 
to be able to receive direct supervision of his/her observation efforts, it 
is possible to receive feedback through other means, such as: 

e Observing a project as someone else is independently, but simul- 
taneously observing the project and comparing the findings for 
accuracy 

e Observing a project and comparing your findings with other evalu- 
ation measures (e.g., questionnaires) for accuracy 

e Observing a project and comparing your findings with the person(s) 
running the project for accuracy 
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• Observing a project and receiving feedback from the penon(t) 
running the project for feedback regarding your style 



• Observing a project and interviewing participants and then comper- 
ing the findings for accuracy 

Sampling It is impossible to record all the behavior that occurs in a particular 

situation. Therefore, it is necessary to make some choices about what is 
to be observed and how it is to be observed. Particularly when using 
direct systematic observation techniques, the intent is more descriptive 
than experimental. Therefore, random sampling is generally not appro- 
priate in descriptive evaluation studies- the kind of study for which you 
are most likely to use observation techniques. The reeeon is that it is 
very unlikely that every person, event, or other element in a given evalua- 
tion situation would have the same likelihood of contributing to the 
study. 

There are three sampling techniques appropriate for use with sys- 
tematic observation. One is "quota-sampling," With this method, it is 
assumed that any organization, group, or situation being studied is made 
up of various people in different roles and the various events that com- 
prise the activity of these people. To determine the sample, you firct find 
out what proportion of the total group is made up of individuals in eech 
role. (You can also find out what proportion of a situation is made up of 
partic jlar types of activities.) Then you can put together your sample by 
including in it persons (or activities) in the proportions in which they 
occur in the organization or situation as a whole. The information you 
gather from the sample may reveal the need for additional categories or 
modifications of those first set up. 

A second sampling approach is the "snowball sample." In this ap- 
proach, you select one or only a few informed persons or do some limited 
exploratory observations. Information gathered through these initial 
contacts generally provide leads to other persons or events to be observed, 
and so on. In this way you can concentrate your attention on the most 
promising leads. 

A third approach to sampling is the "search for expectation." Using 
this technique, you first assume certain relationships between persons, 
events, or variables. For example, you might assume that juniors and 
seniors in high school would be much more interested and active in seek- 
ing out career information than freshman and sophomores are. Using this 
sort of assumption narrows the range of persons and events that one 
would sample. 

Most persons who use systematic observation techniques employ a 
combination of these three sampling techniques in the process of selecting 
who to talk with or listen to and what to observe or read. 

Sampling and justification for the data used must be dealt with 
continually. There is constant concern for sampling such things as: 
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organizational characteristics; events; peoples' knowledge, attitudes, and 
behaviors; roles of individuals; documents; and many other aspects of the 
situation being observed. 



Your As the project director, who you are, how you are perceived, and 

unique what you do, are critical factors in the observation process. Thus, it is 

role very important that you understand your unique role in this process. 

Let us take a look at one way to view the dynamics of a situation to 
be observed. Figure 3 provides a graphic representation of the basic 
elements that play a role in any change situation. There are those who 
are, in a sense, in charge of or advocating or facilitating the development, 
trial, or adoption of a particular career education program, project, or 
activity (the Fs in Figure 3). You may see yourself, or others may see 
you, as part of this group. There is the innovation (I) or change-the 
career education program, project, or activity itself-with its objectives, 
materials, procedures, and so forth. Then there are the consumers (Cs), 
those who are being called upon to use or become involved with the 
change. 

Both the Fs and the Cs have their own perceptions concerning the 
"I." In addition, there will be a large number of interactions that will 
occur between the Fs and the Cs-some about or associated with the 
"I" and others not related. All of these things will take place within a 
social milieu of anticipated and unanticipated factors. 

Figure 4 illustrates three fundamental role positions you might find 
yourself in as an observer. You may be an outsider (rep r ese nte d by a 1 
in Figure 4) or may be perceived to be an outsider. This perception may 
occur even if you have been a part of the group and still feel yourself to 
be. This can happen if you have become a project director within the 
school in which you have been teaching or counseling or have been shifted 
to work with schools in which you have not been a part. You might also 
be within the general context of the situation represented by a 2. For 
instance, you may be an administrator or curriculum coordinator at the 
district level, but not directly involved in the administration of the par* 
ticular career education project. On the other hand, you may be in the 
midst of it all (represented by a 3). You mey be the project director, or 
an assistant project director, or someone who is an actual participant in 
the career education project. 

Whatever the case may be, each role will carry with it different 
expectations on the part of those with whom you must interact to obtain 
information. The role we have most often referred to in this handbook 
is that of 3. 

Another thing that you must determine is whether you see your role 
in the way others see it. If not, serious problems may result as the project 
is planned, implemented and evaluated. 
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Figure 3 

DYNAMICS OF A CHANGE SITUATION 



(F) Facilitators (C) Consumer* 

of Change 



Interactions 
Between or 




(I) Innovation 
or Suggested 
Change 



Figure 4 

YOUR ROLE AS AN OBSERVER 




(F) Facilitators 
of Change 



Interactions 
Between or 
Among (F)&(C) 



(I) Innovation 
or Suggested 
Change 



Observer Roles 

1. External-third party, or perceived outsider 

2. Internal to the "system" but not part of project 

3. Internal to project-possibly an (F) or (C) 



(C) Consumers 




Perceptions 

of (I) by (C) 
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The important thing to remember in reference to this discussion of 
your role in the observation process is that in almost all cases, except 
that of the unobtrusive data collection processes, you have a potential 
influence on the situation. Therefore, you must be diligent in systeme- 
tically checking to see what effect you may be having and make appro- 
priate modifications. 
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CHAPTER 5 



HOW TO ORGANIZE, ANALYZE, AND JUSTIFY 
THE USE OF SYSTEMATIC OBSERVATION DATA 



Introduction 



In the previous four chapters, we have attempted to outline some of 
the basics of collecting systematic observation data. This chapter presents 
specific exercises you can use to develop your systematic observational 
abilities. There are, however, some other tasks for which you must 
prepare if you are to effectively use any data collected through the sys- 
tematic observation techniques discussed thus far in this handbook. 
These tasks are organizing, analyzing, and preparing a justification for the 
inclusion of any observation data in an evaluation report. 



Organizing 
your data 



Data collected through the use of systematic observational tech* 
niques can become rather unwielding unless they are systematically 
recorded and organized. In any attempt to collect systematic observation 
data, you will probably use several people and a variety of techniques. 
As you may have already imagined, systematic observation data like this 
will pile up very quickly. You will need a filing system that holds the 
various systematic observation information and data until they are 
analyzed. This filing system can be organized by type of systematic 
observational report (e.g., observation, interview, or unobtrusive data). 
Or it could be organized by events or segments of the project being 
directed and evaluated. Whatever the basic organization, the best way to 
arrange reports within the categories is chronologically. Patterns of 
behavior may occur over time and it is important to know when what 
happened. 



Analyzing 
your data 



Analysis of systematic observation data should occur periodically 
throughout the duration of the project. In order to obtain the most 
benefit out of the data or to organize yourself for continuing or following 
up leads in the data, you must analyze all data present on a somewhat 
regular basis. Many who work with systematic observation data do so by 
themselves. In a career education project, however, you will probably 
have several persons to assist you in directing and evaluating. If you do 
have assistants working closely on the program, as a group you could 
periodically (e.g., once or twice a month) sit down and gather together all 
the systematic observation data, along with other data you might have 
available and attempt to derive some tentative hypotheses about what is 
happening in the program. It is extremely important not to come to 
conclusions too quickly. One way to put this is "to let the data speak to 
you." Don't read into the data conclusions that you "feel" are emerging. 
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After a good analysis of the data and development of a list of ten* 
tative hypotheses, ask yourself whether there are areas that were left 
unexplored. Then set up a continuation schedule for systematic obser- 
vations, making sure that you follow up on hazy, missing, or conflicting 
information about what appeared to be an important event* Also make 
plans to do systematic observations in areas less explored in order to 
maintain a broad view of the program and not become locked into one 
aspect. Be sure not to discount some things you are observing such as 
department meetings just because nothing related to your program has 
come up recently. You may be surprised to learn that, although they 
were not directly related to your program, some of the discussions did 
impact either positively or negatively on your program. One attitude 
that persons who use systematic observation methods take is that in any 
social setting any aspect or event in the setting in some way affects all 
other aspects of it. Although it may be difficult to see in any single 
observation or even a collection of observations, over time and with 
carefully recorded observations, this fundamental assumption usually 
holds. 



Justifying Many people will say that it is all well and good to use systematic 

systematic observation data for your own work in conducting a program but th|t 

observation there is really no way it can be used objectively in an evaluation report, 

data This is simply not true. As has been pointed out in discussing each of the 

three categories of observation techniques, there are ways of assuring that 
you have information that accurately reflects what did occur. In any 
justification of data used in any evaluation effort, it is important to detail 
the procedures used to collect, organize, and analyze the data. Having 
your systematic observation records well organized and open for inspec- 
tion by those sitting in judgment of the data or report will go a long way 
toward convincing them of the accuracy of the data. 

Another way to convince the reader of the report of the accuracy of 
systematic observational data is to write the report in language that 
reflects what was observed to have happened. Of course, you will also 
have to spell out when it happened and include your reasoning to back 
up your interpretation of the data. 

You will want to avoid writing systematic observation reports as 
if your interpretation is the only possible one. In the presentation of any 
data for evaluative purposes, the conclusions are still the perogative of 
those reading the evaluation report. What you want to do is to provide 
the most accurate description and interpretation of events and outcomes 
as you were able to record them. Anecdotes, references to times, places, 
and records on file will all help convince any reader that the data you pre- 
sent are an accurate and reasonable interpretation of what did occur with 
the program. No "level of statistical significance" is any more convincing. 
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CHAPTERS 



ENHANCING YOUR SYSTEMATIC OBSERVATION SKILLS 7 



Introduction Your success as a project director, an evaluator, or a teacher depends 

in large measure on your ability to observe carefully what is going on 
around you, concisely noting that behavior, analyzing it in relation to 
other observations that have been noted, and then evaluating the informa- 
tion in order to make an appropriate response. Such skills require that 
you be able to separate fact from feeling. In other words, you must be 
able to separate the behavior you observe from the subjective impressions 
you have about that behavior. Both the objective fact and the subjective 
interpretation are important, but it is also important that you be able to 
distinguish between the two and likewise record the two accounts in a 
parallel but separate manner. To do this requires training and practice. 
In this chapter, we hope to provide you with some thinking material and 
exercises that could lead to a better understanding and a capacity to 
actually see and record what goes on in social situations, particularly in 
those settings associated with schools. 

The chapter is divided into four sections. The first deals with im- 
proving basic systematic observation skills. The second, third, and fourth 
sections look at how you can analyze your systematic observations, which 
we hope will alter some of the patterns or "ruts" we all get into when 
doing systematic observation. 

Generally, when we observe, our attention wanders or we only catch 
a part of what is going on, or we get so involved in what is going on we 
begin to take sides and make value judgments about what is being said 
and done. Before going on in this chapter, we challenge you to get into 
a classroom situation, a meeting, or any social event and for about ten 
minutes or even five minutes record all you can about what goes on. 
Try to separate what actually goes on from your feelings about it. Use a 
form similar to Figure 5 to record these observations. In order to 9et it 
all down, make up little "shorthand" notations for particular persons, ac- 
tions, phrases, and words. 



7 Much of the design and basic ideas of this chapter are a condensed version of work done as 
a result of the "Developing Observational Skills: Project in Ethnography in Education" at the 
State University of New York at Buffalo. 



Improving basic 
systematic 
observation 
skills 



Figure 6 

SAMPLE FORM FOR RECORDING OBSERVATIONS 



Narrative (Actually Obaarvad) 


Notts (Ftelings, Imprmions, etc.) 
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Once completed, your first observation! might look like those in 
Figure 6. How do you compere? Do you think that the observations in 
Figure 6 ere good end fairly eccurete? No? You ere right. There is too 
much of e mixture of feelings on the narrative side of the reported situa- 
tion. Look at Figure 7. The observations are much more eccurate when 
recorded in this manner. 

Now look at your f irst observation record. Is it complete? Does it 
contain feelings end impressions on the narrative side? If so, reorganize 
your observation record. If you have en opportunity, heve someone else 
critique your observation record for the separation of objective feet 
from feelings or impressions about the situetion. 

Now do another or even several observations of five or ten minutes' 
length end go beck end analyze them in the seme manner. You will begin 
to get some insight into the way in which you personally look at events. 
It would be helpful if at leest one other person could make the obser- 
vation with you. By comparing observation records, you will see how 
each person tends to look et situations just a little differently. Such 
analyses can help you become a much more eccurate end comprehensive 
observer of human events. After some practice at this level of observa- 
tion, you are reedy to dig deeper into your observational skills. 



Changing your Once you begin to get the basics of observation down, you may 

patterns of begin to realize that you tend to observe certain things ebout e situetion. 

systematic Additionally, you may notice that others heve certain patterns of things 

observation that they observe, which are either similer to or quite different from 

yours. In this section of the chapter, we went to determine whether you 
really do have patterns in your observational reports. 

Go back to some of your previous observation records. Now review 
each observation comment or entry. Ygy can usually categorize these as 
being either an observation of some verbs/ comment, a non-verbal com- 
ment, or some use of space by a person. Using one of your observation 
records, categorize each entry as either verbal, non-verbal, use of space, or 
other. For example, the entries in Figure 7 could be categorized as indi- 
cated in Figure 8. 

You will probably find that most of your observations are of verbal 
behavior as in Table 1 . The second most observed behaviors tend to be 
non-verbal, such as facial expressions, gestures, and body movements or 
positions (e.g., facing, standing, sitting). Beginning observers usually 
place less emphasis on the use of space as pertinent information. 

By periodically analyzing your observational records in this manner, 
you should become more aware of your observational patterns and, if 
necessary, be better able to balance your observations among these rather 
than standard categories of observable actions. Some actions that might 
fall into the "other" category could be coughing, sneezing, or thinking. 
The important thing to strive for here is not the systematic or completely 
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Figure 6 



SAMPLE OBSERVATIONS OF A CLASSROOM 



Narrative (Actually Observed) 



Notes (Feelings, Impressions, etc.) 



1. 



Teacher (T) called 

the roll. 



2. 



Some kids were still talking 
and disrupting the class. 



3. 



Teacher (T) threatens one 



The student was quite defiant. 



student by telling him she 
would give him a bad mark. 

4. T finished the roll. 

5. T ask class to take out 
their career planning 
workbooks. 

6. Most did but some were It seems that that the kids in general 
reluctant. don't like this class. 

7. T asked them to turn to The teacher is somewhat grouchy, 
their homework assignment 

in a rather rough tone of 
voice. 

8. One girl asked if they 
could have extra time to 
get their homework done. 

9. The teacher looked at her 
kind of mean, walked right 
over to her and told her 
"definitely not." 
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Figure 7 

REVISED SAMPLE OBSERVATIONS OF A CLASSROOM 



Narrative (Actually Observed) 

1. Taachar (T) bagan the clan 
by calling the role. 



Many of the students were still 
talking while she called the role. 



Notes (Feelings, Impressions, etc.) 



The class was very disruptive. I could 
hardly hear the names called. 



3. Two students in the back of 
the room were talking ite 
loudly and lightly hitti.v 
other on the arm back ana 
forth across the row. 

4. T calls out specifically to 
one of the boys (Tom) and 
said, "If you don> stop that 
I am going to give you a red 
mark today." 

5. The boys did stop their 
hiring and talking. 

6. Once the roll was finished, 

T asked the class to take cut 
their career planning workbooks 
and turn to the page of 
yesterday's assignment. 

7. Most did while a few continued 
talking or simply sat doing nothing. 

8. One girl spoke up and asked, 
"Can I have more time to 
finish the assignment?" 

9. T walked over, stood next to the 
girl's desk, and looked right at her. 

10. T then said, "It was homework 
and should have been done at 
home and not in class." 



It seems that T gives these "marks" as 
a kind of disciplinary demerit. 



There seemed to be a non-responsiveness 
and even reluctance to get going in this class. 

The teacher looked right at the girl and gave 
her a real mean look. 



The teacher was very brusk in the tone of 
her voice. 

I was beginning to feel very uncomfortable 
in the class. 
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CATEGORIZED ENTRIES FROM 
FIGURE 7-TYPE OF BEHAVIOR 



Entry No. 


Verbal 


Non-Verbal 


Use of Space Other 


1 


X 






2 


X 






3 


X Students 
talking 


X Lightly 
hitting 


X Back of the 
room, back and 
forth across rows 


4 


X 






5 




X 




6 


X 






7 


X 


X 




8 


X 






9 




X Stood 
and looked 


X Walked 
over 


10 


X 
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accurate categorization of your observations, what you are after it to 
increase your awareness of what it is you are concentrating your attention 
on during the observation session. 

If you find an imbalance, you need to ask yourself the question: Are 
there other things going on in those situations I am observing that I am 
not picking up and that are equally or at least reasonably important to 
the total understanding of what is going on? It may be that the most 
important thing to focus on is the non-verbal action and interaction that is 
occurring. In another case, it may be the use of space that is most impor- 
tant. There is no particular virtue in gathering information on all aspects 
of the situation. What will be important in an actual observation/record- 
ing session is that you be aware of what it is you actually want to know 
about the situation and that you concentrate on capturing that. 



Do your values All of us have opinions and hold certain values abou: what is appro- 

gat in the way? priate or inappropriate behavior and what kind of behavior can be ex- 

pected in certain situations. In this segment of the chapter, you will have 
an opportunity to examine your values concerning the behaviors you 
observe. 

Start by obtaining a 15- to 20-minute observation of a classroom. 
Once you have the observation record, analyze it w -' indicating whether 
you feel that the behaviors you observed are "use . ' (you would have 
expected them) or "unusual" (you really would not have expected that 
kind of behavior). Figure 9 gives an example of how a person might 
classify the observations in Figure 7. It will help if you can work with 
one or two other persons who have observed the same situation so that 
you can compare your observations and judgments. 

Over time and among a number of observers you will generally find 
that some people focus more on usual or expected behaviors, whereas 
others focus on unusual or unexpected behaviors. Knowing where you 
stand in terms of your focus is again useful information when you are 
recording observations for evaluative purposes. 

To go one step deeper with this exercise, you could make a further 
judgment concerning the observed behaviors by indicating whether you 
feel the behavior is "very desirable," simply "okay," or "it should be 
changed." Such an analysis could further assist you in clarifying your 
values in reference to what you tend to observe. It seems that some 
observers concentrate on behaviors they believe should be changed. 
Others pay more attention to those behaviors they find agreeable. You 
might think that the judgment of usual or unusual would give you the 
same result as classifying the behaviors as agreeable, okay, or disagreeable. 
In some cases, there may be considerable parallel. In other cases, you 
might judge some behavior as highly unusual but, from your perspective, 
very desirable. In fact, you might be the kind of person that particularly 
looks for those unusual but highly desirable behaviors. 
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CATEGORIZED ENTRIES FROM 
FIGURE 7-USUAL/UNUSUAL BEHAVIORS 


Cntrv No 


Usual 


1 In* 11 im\ 
v'WlMH 


1 


X 




2 


X 




3 




X 


4 


X 




5 


X 




6 


X 




7 


X 




8 


X 




9 




X 


10 


X 





5a 

so 



Ttft it. Find out for yourself. What do you pay attention to? Is 
there evidence in your observations that your selection of behaviors is 
guided by your values or predisposition concerning what you enjoy pick* 
ing out among all the possible observable facts within a given situation? 
You'll never know unless you try it. 



Other factors Another way to analyze your observations is by looking at the fre- 

quency with which you observe certain persons in a group. Or do you 
tend to do most of your observing within a particular section of a room? 
Another thing to look for is whether you tend to observe particular roles 
of individuals. For example, do you consider it unusual or inappropriate 
when a student does the teaching in a class? Do you have particular role 
expectations in your mind, and do these influence the manner in which 
you record events? 

All of these analyses are very important in the development of your 
observation skills. Analyzing your observation patterns in these ways will 
not only enhance your observational abilities, but can also contribute to a 
better understanding of who you are, what you believe, and why. 
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CHAPTER 7 
SUMMARY 



Why use This handbook was designed to be a practical guide to help you, the 

systematic career education project director, undertake your evaluation reaponsibill- 

obsarvatton ties. Realizing that you may not have the time, resources nor expertise 

techniques? to conduct an extensive evaluation, we have introduced three systematic 

observation techniques: 

• observing verbal and non-verbal behavior 

• interviewing 

• gathering data unobtrusively 



By using systematic observation techniques, you should be able to obtain 
information about your project that will help you to: 

• make decisions about the most valuable aspects of the project 

• identify problem areas before they become critical 

• discover unintended outcomes and influences of the project 

• develop a comprehensive picture of what happened and a clue as 
to why it happened 



The checklists Some of the techniques included in this handbook may not be 

entirely new to you. Fur example, the use of interviews and anecdotal 
records are often used. However, you may not be conversant about the 
use of other techniques and how you can systematically plan, implement, 
and use them in evaluating your career education project. Thus, this 
handbook presented information in the step-by-step sequence of planning, 
implementing, and using data from systematic observation techniques. 
In addition, this chapter presents a series of nine checklists for use in 
reviewing the information covered in the handbook. They will also help 
you plan your systematic observation efforts whenever you evaluate your 
project. The checklists will then "walk through" the process of planning, 
implementing and using data from systematic observation techniques by 
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raiting key issues. Incidentally, don't forget the checklists also provide 
references to pertinent sections of the handbook to aid in your review, 
selection, and use of appropriate systematic observation techniques. 



The first step in any evaluation effort is determining what should be 
evaluated and why you want to evaluate it. This, of course, holds true 
when using systematic observation techniques. The information you will 
be collecting and presenting will largely concern the outcomes of your 
project or any portion of it. (Outcomes aren't considered to occur only 
at the end of a project; they can be defined incrementally so that they 
can occur at all logical point! during the conduct of the project.) You 
will be trying to find out whether your project, or parts of it, accom- 
plished the objectives you intended. However, when determining what 
you will evaluate it is important to remember that projects often have 
both anticipated and unanticipated outcomes. That is, while we usually 
evaluate a project based upon the accomplishment of its objectives, we 
should also consider any unanticipated outcomes. If no attempt were 
made to look for unanticipated outcomes, they would simply go un- 
noticed or be so vaguely referred to in impressions from other data 
sources as to be of no value to you. Together, they reveal a more com* 
plete picture of the project's effects. 

In addition to outcomes, there are also anticipated and unanticipated 
influences that can affect the conduct and perhaps outcomes of the proj- 
ect. Both of these influences can be useful in explaining what happened 
as a result of your career education project. 

Used conscientiously and systematically, the techniques discussed 
in this handbook can assist you is determining the effect of anticipated 
influences and to what extent anticipated outcomes occurred. They 
should also provide information leading to the discovery of unanticipated 
outcomes and influences. Now let's see if systematic observation holds 
any promise for your career education evaluation efforts. By completing 
the following checklist, you will identify which outcomes of and influ- 
ences on the project can be evaluated. 



Checklist 1: What Can Be Evaluated? 




Are you interested in: 












Ym 


No 


N/A 


Page Rtfartncvs 


• Anticipated Outcomes? 








5-8 


• Unanticipated Outcomes? 








5 


• Anticipated Influences? 
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• Unanticipated Influences? 
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Systematic 
obMrvttion 
techniques 

• Observing 



Three major systematic observation techniques are available to you 
when evaluating your project. They are observing, interviewing and 
gathering data unobtrusively. 

• Observing is a primary and fundamental technique that under- 
girds all of the others discussed in this handbook. In fact, one could easily 
make the case that all forms of data collection are simply derivatives of 
observation. By observing people, information such as feelings toward the 
project, acceptance or resistance to the project, and student behavioral 
outcomes may be obtained. Several observation techniques are available 
to help systematize your observations: 

• Open-ended observation could be used to keep track of what is 
occurring in the project when little or no structure is being used 
to guide the observation. 

• Interaction analysis is a way of describing what goes on in a learn- 
ing environment or setting by coding the observations by categories 
and preparing a matrix to interpret the observations. 

• Isolated behavior enumeration could be used when you have 
rather specific singular behaviors you desire to enumerate. 

. Mapping actions is an observation approach to recording the move- 
ment of all or certain persons in a group. 

• Critical event recording is simply a means of recording any event 
or incident that may have had potential influence on the project. 

• Diary keeping requires that students, teachers, and others associated 
with your project keep a log or diary of events each day. 

You are ready to determine if any of these techniques are appropriate 
for evaluating your project. Checklist 2 provides a format for making this 
determination. 



Checklist 2: Observation Technique 




Which of these techniques 








are appropriate: 


Yes 


No 


N/A 


P»g« Riftrtncet 


• Open-Ended Observation 








11 


• Interaction Analysis 








12-15 


• Isolated Behavior Enumeration 








16 


• Mapping Actions 








17 


• Critical Event Recording 








18-19 


• Diary Keeping 








20 
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One of the major problems that people have with the collection 
and use of observation data it that such data are potentially biased and 
therefore neither valid nor reliable. However, observation data can be 
collected and verified in such a manner as to render them both valid and 
reliable. Checklist 3 presents some points to be kept in mind in order to 
insure greater confidence in the data you collect. 



Checklist 3; Aseurine Confidence in Observation Dtta 


Will the observer insure that 
he/she: 


Y« 


No 


N/A 


^ - DaIamaam 

rege niierenm 


• Does not become too self* 
conscious about pretending 
to participate on occasion to 
obtain pertinent information 








21 


• Maintains objectivity about 
the project 








21 


* Remembers that observations 
are dependent upon how the 
group perceives the observer 








22 


• Remembers that a person's 
position in a group may affect 
perceptions of any group 
situation 








22 


• Acts in a non-threatening 
way toward members of the 
group being observed 








22 


• Attempts to gain corroboration 
of observation information to 
check the reliability 








22 


• Avoids misinterpretations 








22 


• Does not use diaries nor critical 
events records from untrained 
persons as primary data sources 








22 


• Makes precise category definition 








22 


• Carefully records observations 
according to these categories 








22 
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• Interviewing • Interviewing, or talking to people, it one of the most common 

forms of collecting information. Because it is commonly used, many 
people believe that interviewing is merely asking questions of people. 
Interviews may be used to clarify, confirm, or illustrate. They may also 
be used to obtain opinions and feedback from students, teachers, 
parents, employers, etc. For example, at the beginning of a project, 
interviews can be used to provide you a quick overview of the situation. 
During the project, they can be used to identify areas that need further 
exploration or to confirm information from other sources. When the 
project is completed, they can be used to gather information about the 
impact of certain activities. Interviews are usually classified into two 
types: unstructured (a purposeful discussion between two people with- 
out specific prestructured questions) and structured (a questionnaire 
asked of a respondent either in person or over the telephone). Whether 
you choose to use unstructured and/or structured interviews, you should 
remember that interviewing is more than merely asking questions of 
people. Although asking questions is naturally a part of the interview* 
ing process, the success of an interview does not lie in simply asking 
questions. Obtaining information through interviews requires careful 
planning. You are now ready to determine if either of these techniques 
are appropriate for evaluating your project. Checklist 4 provides a for- 
mat for making this decision. 



Checklist 4: Interview Technique* 




Which of these techniques are 
appropriate? 












Ym 


No 


N/A 


P»gt References 


• Unstructured Interview 
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• Structured Interview 
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As was the case with observation techniques, interview data can 
also be potentially biased. The successful interview requires careful 
planning and control. Confidence in your interviews can be assured in 
several ways. For example, interview data tends to be strongest when 
combined with data from other sources collected through other means. 
Also, the validity of interview data can be judged on whether the infor- 
mation is relevant to the intent of the question and accurate from the 
respondent's perspective. Thus, a number of precautions need to be taken 
so that you will have confidence in your interview. Checklist 5 identifies 
points you should consider in order to obtain more valid and reliable infor- 
mation from your interviews. 
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Checklist 6: /touring Confidence in Interview Data 




Will the interviewer insure 
that he/she: 


Yei 


No 


N/A 


Pegs References 


• Carefully plans and controls 
an interview 








23 


• Uses more than one informed 
person to supply information 
in order to improve the 
reliability of the data 








24 


• Keeps the identify of the 
informed persons anonymous 








24 


• IHantifiM th* data oeth«rad 

as primary source, secondary 
source or hearsay 








24 


a Obtains information on 
sentiments about a situation 
and a*tAriat*H intftroareonal 
action 








24 


• Goes step by step through 
particular situations or evei.^s 
after the general section of 
the interview 










• Probes the interviewee in 
a variety of ways 








25 


• Pins down events, people and 
times 








25 


• Assures the interviewee that 
the information will be held 
in strict confidence 








25 


• Combines information from 
the interview with other 
sources to increase validity 
and reliability 








26 


• Records the interview data 
accurately according to the 
categories of responses 








27 


• Checks consistency of 

responses across interviewers 








27 
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• Unobtrusive techniques art those data collection techniques that 
do not require direct contact with the individual. Sevtral techniques are 
available, including physical traces, artifacts, records and indices. Check- 
list 6 provides a format to help you select the appropriate unobtrusive 
techniques for your project. 



Checklist S: Unobtrusive TeohrOquee 




Which of these techniques 
•rt appropriate? 


Yes 


No 


N/A 


»ege References 


• Physical Tracts 








27 


• Artifacts 








2728 


• Rtcords 








28 


• Indices 








28 



One can also have confidence in data collected unobtrusively in that 
it provides leads and corroborates other self-report or observation data. 
It is not often that unobtrusive evidence alone is sufficient. As with the 
other systematic observation techniques discussed, certain precautions 
should be taken when you gather data unobtrusively. Thus, there are 
several points that should be addressed in order to increase your con- 
fidence in this data. 



Checklist 7: Assuring Confidence in Unobtrusive Data 


The evaluator will insure 
he/she: 


Yet 


No 


N/A 


Page References 


• Determines who participated 
in the avent studied 








29 


• Determine the period of 
time over which the event 
occurred 








29 


e Corroborates the data by 
using other techniques to 
measure the event 








29 
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• Gathering data 
unofatruiivaty 



Things to When using any systematic observation technique you should take 

consider when into account the many things that potentially affect the evaluation, 

using systematic These include data collection; being accepted; sources of tension; training 

observation and practice sampling and your role. These points were discussed in the 

techniques handbook and are presented in Checklist 8. By reviewing these points 

as you plan your systematic observation, you will gather more quality 

information. 





Checklists: Thin* to Consider When Using 

Systematic Observation Techniques 


H 


ivt you considered: 




No 


N/A 


Page References 


• 


Will tht privacy rights of 
individuals bs protected? 








31 


• 


Will the observer's identify 
be revealed? 








31*32 


• 


Will the observer's identity 
be concealed? 








31 32 


• 


Will the data collected 
be reliable? 








32 


• 


Will the data collected 
be validated? 








32 


• 


Will the observer be accepted 
by the group? 








3233 


• 


Wilt political or interpersonal 
relationships affect tht 
systematic observation? 








34 


• 


Will the observer have an 
understanding of systematic 
observation techniques? 








34 


• 


Will the observer have practice 
with systematic observation 
techniques? 








34 


• 


Will sampling methods be used? 








3536 


• 


Will the project director be 
certain of hi^er role in the 
systematic observation 
process? 








36-39 
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How to organize, 
analyze, and 
justify tht us* 
of systematic 
observation data 



In addition to planning and collecting systematic observation infor- 
mation and data, you must also prepare to use it effectively. These tasks 
are organizing, analyzing and preparing a justification for the inclusion of 
any systematic observation data in an evaluation report. This handbook 
raised several points that affect organizing, analyzing, and using systematic 
observation data. They are highlighted in Checklist 9. 



Checklitt9: How to Organize, Analyse, snd 
Justify tht Un of Systematic 
Observation Data 




Have you considered: 


Ym 


No 


N/A 


Page Reference! 


• Designing a filing system to 
hold systematic observation 
and information data until 
they are analyzed? 








41 


• Analyzing data on a regular 
basis rather than after they 
are all collected? 








41 


• Generating hypotheses from 
the information and data? 








41*42 


• Scheduling additional sys- 
tematic observations to 
clarify hazy, missing, or 
conflicting information? 








42 


• Writing the procedures 
used to collect, organize, 
and analyze data m the 
evaluation report? 








42 


• Writing the evaluation 
report in language that 
reflects what was observed 
and why it was interpreted 
as such? 








42 
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It's up to you! The information presented ir this handbook was designed to help 

you systematize what you do daily-observe. By using the techniques 
discussed in this handbook the information you obtain will move from 
"gut feelings" to evaluation information that is a part of your overall 
evaluation plan. Systematic observation can hold a special place in your 
evaluation plan because it allows you to obtain evaluation information 
as an activity occurs at a relatively low cost. Becoming a "trained" 
observer only takes practice. The more you practice, the more infor- 
mative your systematic observations will become. 

The evaluation techniques you choose for your project will vary 
based upon your information needs and resources. We have tried to 
introduce you to some techniques which are based upon observation 
rather than being based solely upon paper and pencil tests. You can use 
one systematic observation technique or a combination of systematic 
observation techniques, the choice is yours. 

The steps in the systematic observation process are not difficult. 
A quick review of the checklists should confirm this. But, the rewards 
are plentiful. With a little advanced planning and a little practice, you can 
use systematic observation techniques which require little time, staff and 
money to gather a wealth of information about your project. Well, it's 
now up to you Happy Observing! 
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